{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Enviroment: \n",
    "Open AI gym [CartPole v0](https://github.com/openai/gym/wiki/CartPole-v0)\n",
    "\n",
    "\n",
    "\n",
    "### Observation\n",
    "\n",
    "Type: Box(4)\n",
    "\n",
    "| Num  | Observation          | Min      | Max     |\n",
    "| ---- | -------------------- | -------- | ------- |\n",
    "| 0    | Cart Position        | -2.4     | 2.4     |\n",
    "| 1    | Cart Velocity        | -Inf     | Inf     |\n",
    "| 2    | Pole Angle           | ~ -41.8° | ~ 41.8° |\n",
    "| 3    | Pole Velocity At Tip | -Inf     | Inf     |\n",
    "\n",
    "### Actions\n",
    "\n",
    "Type: Discrete(2)\n",
    "\n",
    "| Num  | Action                 |\n",
    "| ---- | ---------------------- |\n",
    "| 0    | Push cart to the left  |\n",
    "| 1    | Push cart to the right |\n",
    "\n",
    "Note: The amount the velocity is reduced or increased is not fixed as it depends on the angle the pole is pointing. This is because the center of gravity of the pole increases the amount of energy needed to move the cart underneath it\n",
    "\n",
    "### Reward\n",
    "\n",
    "Reward is 1 for every step taken, including the termination step\n",
    "\n",
    "### Starting State\n",
    "\n",
    "All observations are assigned a uniform random value between ±0.05\n",
    "\n",
    "### Episode Termination\n",
    "\n",
    "1. Pole Angle is more than ±12°\n",
    "2. Cart Position is more than ±2.4 (center of the cart reaches the edge of the display)\n",
    "3. Episode length is greater than 200\n",
    "\n",
    "### Solved Requirements\n",
    "\n",
    "Considered solved when the average reward is greater than or equal to 195.0 over 100 consecutive trials\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. gym enviroment setup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 0.01610188, -0.02496384, -0.0061698 ,  0.01667973])"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import gym\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "env = gym.make(\"CartPole-v0\")\n",
    "env.reset()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Q Table setup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "LEARNING_RATE = 0.5\n",
    "DISCOUNT = 0.95\n",
    "EPISODES = 50000\n",
    "SHOW_EVERY = 1000\n",
    "Q_TABLE_LEN = 150"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "def sigmoid(x):\n",
    "  return 1 / (1 + np.exp(-x))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/hongtao/anaconda3/envs/spinningup/lib/python3.6/site-packages/ipykernel_launcher.py:2: RuntimeWarning: overflow encountered in exp\n",
      "  \n"
     ]
    }
   ],
   "source": [
    "DISCRETE_OS_SIZE = [Q_TABLE_LEN] * (len(env.observation_space.high))\n",
    "\n",
    "\n",
    "observation_high = np.array([env.observation_space.high[0],\n",
    "                    Q_TABLE_LEN*sigmoid(env.observation_space.high[1]),\n",
    "                    env.observation_space.high[2],\n",
    "                    Q_TABLE_LEN*sigmoid(env.observation_space.high[3])])\n",
    "\n",
    "observation_low = np.array([env.observation_space.low[0],\n",
    "                    Q_TABLE_LEN*sigmoid(env.observation_space.low[1]),\n",
    "                    env.observation_space.low[2],\n",
    "                    Q_TABLE_LEN*sigmoid(env.observation_space.low[3])])\n",
    "\n",
    "discrete_os_win_size = (observation_high - observation_low) / DISCRETE_OS_SIZE\n",
    "\n",
    "# q_table = np.random.uniform(low=0, high=1,\n",
    "#                             size=(DISCRETE_OS_SIZE + [env.action_space.n]))\n",
    "\n",
    "q_table = np.zeros((DISCRETE_OS_SIZE + [env.action_space.n]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(150, 150, 150, 150, 2)"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "q_table.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Decay epsilon "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "epsilon = 1  # not a constant, qoing to be decayed\n",
    "START_EPSILON_DECAYING = 1\n",
    "END_EPSILON_DECAYING = EPISODES//2\n",
    "epsilon_decay_value = epsilon/(END_EPSILON_DECAYING - START_EPSILON_DECAYING)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Help functions "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_discrete_state (state):\n",
    "    discrete_state = (state - observation_low) // discrete_os_win_size\n",
    "    return tuple(discrete_state.astype(int))\n",
    "\n",
    "def take_epilon_greedy_action(state, epsilon):\n",
    "    discrete_state = get_discrete_state(state)\n",
    "    if np.random.random() < epsilon:\n",
    "        action = np.random.randint(0,env.action_space.n)\n",
    "    else:\n",
    "        action = np.argmax(q_table[discrete_state])\n",
    "    return action"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Rewards Recorder setup "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "ep_rewards = []\n",
    "aggr_ep_rewards = {'ep':[],'avg':[],'min':[],'max':[]}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5. Train the Agent "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "episode: 0\n",
      "episode: 1000\n",
      "episode: 2000\n",
      "episode: 3000\n",
      "episode: 4000\n",
      "episode: 5000\n",
      "episode: 6000\n",
      "episode: 7000\n",
      "episode: 8000\n",
      "episode: 9000\n",
      "episode: 10000\n",
      "episode: 11000\n",
      "episode: 12000\n",
      "episode: 13000\n",
      "episode: 14000\n",
      "episode: 15000\n",
      "episode: 16000\n",
      "episode: 17000\n",
      "episode: 18000\n",
      "episode: 19000\n",
      "episode: 20000\n",
      "episode: 21000\n",
      "episode: 22000\n",
      "episode: 23000\n",
      "episode: 24000\n",
      "episode: 25000\n",
      "episode: 26000\n",
      "episode: 27000\n",
      "episode: 28000\n",
      "episode: 29000\n",
      "episode: 30000\n",
      "episode: 31000\n",
      "episode: 32000\n",
      "episode: 33000\n",
      "episode: 34000\n",
      "episode: 35000\n",
      "episode: 36000\n",
      "episode: 37000\n",
      "episode: 38000\n",
      "episode: 39000\n",
      "episode: 40000\n",
      "episode: 41000\n",
      "episode: 42000\n",
      "episode: 43000\n",
      "episode: 44000\n",
      "episode: 45000\n",
      "episode: 46000\n",
      "episode: 47000\n",
      "episode: 48000\n",
      "episode: 49000\n"
     ]
    }
   ],
   "source": [
    "for episode in range(EPISODES):\n",
    "    # initiate reward every episode\n",
    "    ep_reward = 0\n",
    "    if episode % SHOW_EVERY == 0:\n",
    "        print(\"episode: {}\".format(episode))\n",
    "        render = True\n",
    "    else:\n",
    "        render = False\n",
    "\n",
    "    state = env.reset()\n",
    "    done = False\n",
    "    while not done:\n",
    "        action = take_epilon_greedy_action(state, epsilon)\n",
    "\n",
    "        next_state, reward, done, _ = env.step(action)\n",
    "\n",
    "        ep_reward += reward\n",
    "\n",
    "        # if render:\n",
    "        #     env.render()\n",
    "\n",
    "        if not done:\n",
    "\n",
    "            td_target = reward + DISCOUNT * np.max(q_table[get_discrete_state(next_state)])\n",
    "\n",
    "            q_table[get_discrete_state(state)][action] += LEARNING_RATE * (td_target - q_table[get_discrete_state(state)][action])\n",
    "\n",
    "\n",
    "\n",
    "        state = next_state\n",
    "\n",
    "    # Decaying is being done every episode if episode number is within decaying range\n",
    "    if END_EPSILON_DECAYING >= episode >= START_EPSILON_DECAYING:\n",
    "        epsilon -= epsilon_decay_value\n",
    "\n",
    "    # recoard aggrated rewards on each epsoide\n",
    "    ep_rewards.append(ep_reward)\n",
    "\n",
    "    # every SHOW_EVERY calculate average rewords\n",
    "    if episode % SHOW_EVERY == 0:\n",
    "        avg_reward = sum(ep_rewards[-SHOW_EVERY:]) / len(ep_rewards[-SHOW_EVERY:])\n",
    "        aggr_ep_rewards['ep'].append(episode)\n",
    "        aggr_ep_rewards['avg'].append(avg_reward)\n",
    "        aggr_ep_rewards['min'].append(min(ep_rewards[-SHOW_EVERY:]))\n",
    "        aggr_ep_rewards['max'].append(max(ep_rewards[-SHOW_EVERY:]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Text(0, 0.5, 'Rewards')"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYwAAAEGCAYAAAB2EqL0AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nOzdd3hUVfrA8e9J76QnpEAKJZAECAkgRUCxgAXBwooNdFfsbdWfba27uuqK2NYuIhbWgghioygoSgu9pUKAFNIzSUjPnN8fdyYESZmUyUwm5/M8eWbmzi3vBHLfOV1IKVEURVGU9thZOgBFURSld1AJQ1EURTGJShiKoiiKSVTCUBRFUUyiEoaiKIpiEgdLB9AV/v7+MiIiwtJhKIqi9Co7duwoklIGdPS4Xp0wIiIiSE5OtnQYiqIovYoQ4mhnjlNVUoqiKIpJVMJQFEVRTKIShqIoimKSXt2G0ZL6+nqys7OpqamxdChm4+LiQlhYGI6OjpYORVGUPsTmEkZ2djaenp5EREQghLB0ON1OSklxcTHZ2dlERkZaOhxFUfoQs1VJCSHChRC/CCEOCiEOCCHuMWz3FUKsFUKkGx59DNuFEOI1IUSGEGKvEGJ0Z65bU1ODn5+fTSYLACEEfn5+Nl2CUhTFOpmzDaMBuF9KORw4C7hDCDEceBhYL6UcDKw3vAaYAQw2/CwA3urshW01WRjZ+udTFMU6ma1KSkqZB+QZnlcIIQ4BocBlwFTDbh8BG4CHDNuXSm2+9S1CCG8hRH/DeRQrU1RdRPKJZKZHTu/xaxdXF7Pm6BqmR0zHx8Wn3f2r6qtYlbmKouqiHohOUXrGIJ9BTI/o2b+/HmnDEEJEAAnAViCoWRI4AQQZnocCx5sdlm3YdlrCEEIsQCuBMGDAALPFrLTty7QveXP3m8T5xxHmGdZj15VS8shvj7A5bzOLdizimphrmBc7r8XEcbL+JMtSlvHRgY8oqy1DoEpmiu2YHjHd9hKGEMIDWA7cK6Usb16dIqWUQogOreAkpXwXeBcgKSlJrf5kIdkV2QDsLNjZownjq/Sv2Jy3mZvjbya7IpvF+xfzWcpnzI2Zy7zYefi6+J6RKCaFTuK2kbcxImBEj8WpKLbIrAlDCOGIliw+lVJ+bdicb6xqEkL0BwoM23OA8GaHhxm29UqzZs3i+PHj1NTUcM8996DX68nMzOQ///kPAEuWLCE5OZk33niDf/7zn3zyyScEBAQQHh5OYmIiDzzwgIU/QduMCWNH/g5mRs/skWvmVObw0vaXGNd/HHcl3IUQgltG3sI7e9/hw/0fsixlGecNOI9fc35FV6vj7NCzuW3kbcQHxPdIfIpi68yWMIRWlPgAOCSlfLnZW6uAecDzhseVzbbfKYT4HzAO0HW1/eLpbw9wMLe8K6c4w/AQL568NLbd/RYvXoyvry/V1dWMGTOG9evXM3HixKaE8fnnn/PYY4+xfft2li9fzp49e6ivr2f06NEkJiZ2a8zmkF15KmH0BL3U8+TvTwLwzIRnmhr+o72jeXHyi9w64lbe2fsO3x35rqlEEecf1yOxKUpfYc4SxkTgemCfEGK3YdujaIniCyHEX4GjwBzDe98DFwEZQBVwoxljM7vXXnuNFStWAHD8+HGOHDlCVFQUW7ZsYfDgwaSkpDBx4kReffVVLrvsMlxcXHBxceHSSy+1cOTtq22spaCqAG9nb46WH6Wough/V3+zXvPL1C/ZemIrT4x/ghCPkDPej/KO4oXJL/D82c+rXmSKYibm7CW1CVptZZzWwv4SuKM7YzClJGAOGzZsYN26dWzevBk3NzemTp1KTU0NV199NV988QUxMTHMnj27197Yciq1msKLIi/is5TP2JG/gwsjLjTb9bIrslm4YyHj+4/nysFXtrlvb/2dKkpvoOaSMgOdToePjw9ubm6kpKSwZcsWAGbPns3KlStZtmwZV199NQATJ07k22+/paamhsrKSlavXm3J0E2SU6EljPMGnoerg6tZq6X0Us8TfzyBvbDnmYnPqISgKBZkc1ODWIPp06fz9ttvM2zYMIYOHcpZZ50FgI+PD8OGDePgwYOMHTsWgDFjxjBz5kxGjBhBUFAQ8fHx9OvXz5Lht8vYfhHhFcHIgJFmTRifp37O9hPbeXrC0wS7B5vtOoqitE8lDDNwdnbmhx9+aPG9lkoQDzzwAE899RRVVVVMnjzZ6hu9cypycLZ3xt/Vn8SgRN7c/Sa6Wh39nLs30R0vP86iHYuYGDqR2YNmd+u5FUXpOFUlZQUWLFjAqFGjGD16NFdccQWjR3dqGq0ek12ZTahHKEIIEoMSkUh2F+xu/8AOem/fewgET41/SlVFKYoVUCUMK/DZZ59ZOoQOyanMaRqsF+8fj4OdAzvydzAlfEq3XmdXwS7G9h+rqqIUxUqoEobSIVJKsiuyCfPQEoaLgwvx/vHd3o6hq9WRVZ7FCH81OltRrIVKGEqH6Gp1VNZXEuoR2rQtMSiRg8UHqaqv6rbr7C/aD6BGaSuKFVEJQ+kQ4xiM5vNHJQYl0iAb2Fu0t9uus7doLwJBnJ8ara0o1kIlDKVDjldqEwo3L2GMChiFnbDr1mqpfYX7iPaOxsPJo9vOqShK16iEYSGrVq3i+eeft3QYHWYctNe8hOHh5MFQn6HdljCklOwr2ke8v6qOUhRronpJWcjMmTOZObNnZnntTtmV2fg4++Du6H7a9sSgRL5M+5K6xjqc7J26do2KbMpqy1T7haJYGVXCMIOsrCxiYmKYP38+Q4YM4dprr2XdunVMnDiRwYMHs23bNpYsWcKdd94JwPz587n77ruZMGECUVFRfPXVVxb+BK3Lrshucf2LpKAkahtrOVh8sMvX2FO0B0D1kFIUK2PbJYwfHoYT+7r3nMHxMKP9qqSMjAy+/PJLFi9ezJgxY/jss8/YtGkTq1at4rnnnmPWrFmn7Z+Xl8emTZtISUlh5syZXHll25PsWUpOZQ6xfmdO6pgQlABAcn4yowJHdeka+wr34ergSrR3dJfOoyhK91IlDDOJjIwkPj4eOzs7YmNjmTZtGkII4uPjycrKOmP/WbNmYWdnx/Dhw8nPz+/5gE3QqG8krzKvxRKGr4svUf2iuqUdY1/RPmL9YnGws+3vM4rS29j2X6QJJQFzcXZ2bnpuZ2fX9NrOzo6GhoY299dmerc++VX5NMiGpkF7f5YYlMgPR36gUd+IvZ19p65R11hHSkkK1w2/riuhKopiBqqEoZjMuCxrqGdoi+8nBiVSWV9JWmlap6+RUpJCvb5etV8oihVSCUMxWdOgvTZKGNC1ZVv3FWltTqpLraJYH3Ou6b0YuAQokFLGGbZ9Dgw17OINlEkpRwkhIoBDQKrhvS1SylvNFZu5RUREsH///qbXS5YsafG9+fPnn/E+QGVlpblD7JTjFcexF/atTgYY7B5MqEcoO/J3dLpKaU/hHgLdAglyD+pKqIqimIE52zCWAG8AS40bpJR/MT4XQiwEdM32z5RSdq17jWJW2ZXZBLsHt9kYnRiUyKacTUgpOzUl+b7Cfao6SlGslNmqpKSUvwIlLb0ntDvJHGCZua6vdL+cypxWq6OMEoMSKakp4bDucIfPX1JTQnZlthqwpyhWylJtGGcD+VLK9GbbIoUQu4QQG4UQZ7d2oBBigRAiWQiRXFhYaP5IlSatDdprbkLIBASCn7J+6vD5jTPUqhKGolgnSyWMuZxeusgDBkgpE4C/A58JIbxaOlBK+a6UMklKmRQQENADoSoAVfVVlNSUtJswgt2DGdd/HCszVqKX+g5dY2/hXuyFPcP9hnclVEVRzKTHE4YQwgG4HPjcuE1KWSulLDY83wFkAkN6OjaldcYeUs1nqW3NrEGzyD2ZS/KJ5A5dY1/RPgZ5D8LN0a1TMSqKYl6WKGGcB6RIKbONG4QQAUIIe8PzKGAw0PFKcMVsjGMw2mvDADh3wLl4OHrwTcY3Jp9fL/XsK9yn2i8UxYqZLWEIIZYBm4GhQohsIcRfDW9dzZmN3ZOBvUKI3cBXwK1SyhYbzBXLaCphtDJorzlXB1emR05n7dG1VNaZ1kU4qzyLivoK1X6hKFbMbN1qpZRzW9k+v4Vty4Hl5opF6brsymzcHNzwcfYxaf9Zg2bxVdpXrDm6hssHX97u/vsK1YA9RbF2aqS3GZgyvfm2bdsYP348CQkJTJgwgdRUbcziokWLuOmmmwDYt28fcXFxVFV131rZnZVTkUOoZ6jJYytG+I8gwiuClRkrTdp/X9E+3B3diewX2ZUwFUUxI5uefPCFbS+QUpLSreeM8Y3hobEPtbtfe9ObL126lN9++w0HBwfWrVvHo48+yvLly7nnnnuYOnUqK1as4Nlnn+Wdd97Bzc3yjcDZldmEe4abvL8QglmDZvHKzlc4Wn6UgV4D29x/b+Fe4vzjOj1poaIo5qdKGGbS3vTmOp2Oq666iri4OO677z4OHDgAaLPZLlmyhOuvv54pU6YwceJEC38SbfbcnMqcdrvU/tml0ZdiJ+zaLWVUN1STXpqu2i8UxcrZdAnDlJKAubQ3vfnjjz/OOeecw4oVK8jKymLq1KlN+6enp+Ph4UFubm5Ph92i4ppiqhuqTepS21ygWyATQiawKnMVd4y6o9XSw6HiQzTIBtV+oShWTpUwLESn0xEaqt2Am08+qNPpuPvuu/n1118pLi62iuVajV1qO1IlZTRr0Czyq/LZmre11X2aZqhVXWoVxaqphGEh//d//8cjjzxCQkLCaQsq3Xfffdxxxx0MGTKEDz74gIcffpiCggILRtqxQXt/NjV8Kl5OXnyT2fqYjD2FewhxD8Hf1b/TMSqKYn42XSVlKaZOb56WdmqhoX/9618ALF68uGlbeHg4GRkZZo62fcYSRohHSIePdbZ35qLIi1iRsYLyunK8nE7N+KKr1fGf7f9h7dG1XDH4im6LV1EU81AlDKVdOZU5+Lv64+rg2qnjZw2aRW1jLT8e+bFp28bjG7l85eWsPryam+Nv5tFxj3ZXuIqimIkqYSjtyq7MNmlKkNYM9xvOIO9BrMxYyYURF/LCthf49vC3DPIexGvTXiPWL7Ybo1UUxVxsMmF0dvGe3kJK2aPXy67Iblp+tTOMYzJeSn6Jmd/MpLy2nFtG3MItI27B0d6xGyNVFMWcbK5KysXFheLi4h6/qfYUKSXFxcW4uLj0yPXqG+vJr8rvVIN3cxdHXYyrgysBrgF8dvFn3Jlwp0oWitLL2FwJIywsjOzsbGx5cSUXFxfCwjpfRdQReSfz0Et9hwft/Zm/qz8/XP4DXs5eONqpRKEovZHNJQxHR0ciI9V8RN0lu1LrIdXVEgaAn6tfl8+hKIrl2FyVlNK9ujJoT1EU26IShtKm7MpsHOwcCHBVy+EqSl+nEobSpuyKbEI9QtUssoqiqIShtC29NJ3oftGWDkNRFCtgziVaFwshCoQQ+5tte0oIkSOE2G34uajZe48IITKEEKlCiAvNFZdiupqGGo5VHGOwz2BLh6IoihUwZwljCTC9he2LpJSjDD/fAwghhqOt9R1rOOZNIYSqA7GwTF0meqlniM8QS4eiKIoVMFvCkFL+CpSYuPtlwP+klLVSyiNABjDWXLEppkkvTQdQJQxFUQDLtGHcKYTYa6iy8jFsCwWON9sn27DtDEKIBUKIZCFEsi0PzrMG6aXpONs7M8BzgKVDURTFCvR0wngLiAZGAXnAwo6eQEr5rpQySUqZFBCgunqaU1ppGtHe0aqHlKIoQA8nDCllvpSyUUqpB97jVLVTDtB8ZFiYYZtiQeml6Qz2VtVRiqJoejRhCCH6N3s5GzD2oFoFXC2EcBZCRAKDgW09GZtyuuLqYoprilWDt6IoTcw2l5QQYhkwFfAXQmQDTwJThRCjAAlkAbcASCkPCCG+AA4CDcAdUspGc8VmrR7+7WGmhE1hRuQMS4dCeplq8FYU5XRmSxhSyrktbP6gjf2fBZ41VzzWrqahhu8Of0dJdYlVJIy0Em35WFXCUBTFSI30thJ5J/MA2FO4hwZ9g4Wj0UoYvi6+aoZZRVGaqIRhJXIrcwGoaqhqGv9gSeml6ao6SlGU06iEYSVyKk91CttZsNOCkUCjvpGMsgxVHaUoymlUwrASOZU5ONg5EOgWyO6C3RaN5XjFcWoba1WXWkVRTmNzK+71VrmVufR370+sXyw7C3YipUQIYZFY0koNDd6+qoShKMopqoRhJXIrcwn1CCUhMIGCqoKmRnBLSC9Lx07YqWnNFUU5jUoYViKnMqcpYQDsKthlsVjSS9MZ4DkAFwcXi8WgKIr1UQnDCtQ01FBcU0yIRwiDfQbj7uhu0YSRVpqmekj1QQ2Nev7IKOKjP7I4mFuOlNLSIbWprkFPRkEFJ2st3w29r1BtGFYg96TWpTbEIwQHOwdG+I+wWMKoqq8iuyKbS6Mvtcj1lZ5V36hnc2YxP+zP46cD+ZScrGt6L9DTmclDApgyJICzB/vj7eZk1lh+TSvkky1H8XBxIMDDGX8PZ/w9nfD3cMbLxZHjpVWk5VeSnl9BWn4FWcVVNOolbk72TI8N5vLRYYyP9sPezjJtf32BShhWIKdC61Ib6qHN6J4QmMBbe96ioq4CTyfPHo0loywDiVRdaq1cQ6Oe1PwKdh4rY9fRUg7klhPu68rYSF/GRfoRG+KFg/2ZFQh1DXoyCytJPVHB7xlFrDmYj666Hncne6YNC+Ki+GCG9+/HliPFbEwrZM2BE3y1Ixs7AXGh/fD3cMbVyR43R3tcnbQfdycHJg7yZ/QA70511Gho1PPy2jTe3JBJoKczjvZ2FFbWUtegP2NfIWCgrxuDgzyZHhdMhJ87O4+VsnpvHl/vyiHYy4VZCaFcPjqUIUE9+7fTF6iEYQWMg/ZC3EMASAhKQCLZU7iHSaGTejQW46DBId4qYVibQ3nlfLc3jx1HS9mTXUZVnTbdmr+HE7Eh/cgsPMm6QwUAuDvZkxjhy7hIXwBSTlSQeqKcw4UnadBrVU2eLg6cPzyIGXH9OXuwPy6Op6axH+DnxpykcBoa9ezJ1rExrZDtR0ooqKihqq6RmrpGquobqaprpK5Bu+HHhnhxw/iBzBwZiquTaVPi55ZVc/eyXSQfLeXqMeE8eWksrk72SCmpqG2gqKKWwopayqrrCfV2ZVCgx2lxAlyVpB23/lABX+/M5r3fDvP2xkziQr24cnQYM0eF4uveeumovlHP9qwS0vMruSi+PwGezqb/o/QxwtrrKduSlJQkk5OTLR1Gl72842U+OfgJydclYyfsqKqvYsKyCfw1/q/clXBXj8by763/ZkXGCrZcswU7oZq4LK2sqo6Vu3P5csdx9ueUY28nGN7fi9EDvBk90IfRA3wI83Ft+mZfUF7D1iMlbD1SzLYjJaTlVwIQ6u1KTLAnQw0/McFeRAW449hCKaSjTtY28M3uHJb+cZTU/Ar6uToyJymM684ayEA/91aP+zkln79/sYf6Bj3PXR7PZaNaXDOtw4oqa1m1O5evd2WzP6ccR3vBuTGBXDE6jHNiAnG0t0NXXc/GtELWHcxnQ2oB5TVaO4iroz03TBjILZOj20wyvZ0QYoeUMqnDx6mEYXkPbHyAlJIUVs9e3bTtL6v/grujO4svXNyjsdz0003UNtTy6cWf9uh1lVMa9ZJf0wv5KjmbtQfzqWvUM7y/F3OSwrhsVCg+HbiRlVXVYWcn8HJxNGPEGikl246UsHTzUX48cAK9lAwN8iTc140wH1dCvV0J89Ger9ydw3u/HWF4fy/euCaBqAAPs8SUcqKc5TuyWbErl6LKWvzcnYgO8GDnsVIa9BI/dyfOiQnkvGFBDPB1491fM1m5Jxc3R3tunBjJ386ONHvbjSV0NmGoKikrkFuZ21QdZZQQmMDytOXU6+txtDP/Hztof/DppelMGzCtR66nnOl4SRV//2I327NK8XFz5JpxA7gqKYzYkH6dOl9P3uyEEIyL8mNclB8ndDX8b/sx9mXrOFZcxe8ZRU1VaEbXnzWQxy4edkYVU3eKCfbisYuH89D0GC0J78jmWEkVN0+O4rxhgYwK9zmtkfyVqxO445xBvLI+nTd+yeCjP7K4aVIkt06JNrmazZaphGEFcipzOCf8nNO2JQQm8OmhT0ktSSXOP65H4iisLqSstkx1qbUAKSUrduXwxMoDCOCFK+KZlRCKs0PvvEkF93Ph3vNOtYNJKSmrqie7tJrs0ip83Z0YF9VzMyE72NtxbkwQ58YEtbvv4CBP/nvNaO46t5xX1qbz6vp0fk4p4P15SQR59e2xSSphWFhVfRUlNSVNPaSMjAP4dubv7LGE0dTgrXpI9aiyqjoe+2Y/3+3NY2yELwvnjCTc183SYXUrIQQ+7k74uDsRH9a50lJPiwn24u3rE1l/KJ+7l+3isjd+5/15ScSF9lz8afkVlJ6sw9fdCW83J3zcHFvs/dZTzLni3mLgEqBAShln2PYf4FKgDsgEbpRSlgkhIoBDQKrh8C1SylvNFZs1MU4B8ueEEegWSKhHKLsLd3MDN7R4rK5Wx4HiAyQGJeJs3/WeHU1zSKmE0a1ST1RQWFGLv6cTAR7O+Lg5YWeoBvkjo4i/f7GHospaHrxwKLdOiVbjCKzMtGFBfHXbBP66ZDtXvb2Z1+YmcP7w9ksqXbUpvYjrPth6xnYvFwd83Z24MDaYRy4aZvY4mjNnCWMJ8AawtNm2tcAjUsoGIcQLwCPAQ4b3MqWUo8wYj1UyTmse4hFyxnsJgQlszt3c4kSE9fp67vr5LnYV7MLd0Z0pYVO4IOICJoVO6nTySC9NJ9A1kH7OveMbYG9Q16Dnyrf/oKLm1GhkezuBr7sTfu5OpOZXEOnnzte3T2BEmLcFI1XaMqy/F9/cOZGbl+5gwcfJPDIjhpvPjjLbBKFlVXU88OUeogPceWpmLGVV9ZRW1VF6UnssOVlnke6/5lyi9VdDyaH5tjXNXm4BrjTX9XsL4xiMP5cwQEsYqw+vJrsim3Cv8NPee33X6+wq2MXtI2/nRNUJ1h9bz/dHvsfNwY0p4VOYETGDqeFTO/QfOq00jcG+qv2iOyVnlVBR08DDM2II83GlqKKWoso6CitqKaqsZfKQAO49bzBuTqp22NoFerrw+YKzuP+LPTz3fQqHC0/yzGVxODl0bxWRlJLHvtlPUWUt78+b2KNVYO2x5P/Sm4DPm72OFELsAsqBf0gpf2vpICHEAmABwIABA8wepLnlVubiZOfU4lKoTRMRFu46LWFsPL6RD/d/yFVDruK2UbcB8I+z/sH2vO2sObqG9cfW88ORHxgXPI6nJjxFmGdYu3HU6+s5rDvMhJAJ3fTJFIANaYU42guuP2sg7s4qKfR2Lo72vD43gagAd17/OYP9uTqenRXPyPDuKx1+szuH7/bm8eCFQ60qWYCFJh8UQjwGNADGzv55wAApZQLwd+AzIYRXS8dKKd+VUiZJKZMCAgJ6JmAzyqnMIcQjpMVBctHe0Xg6ebIz/9QKfLmVuTy66VFifGN4aOxDTdsd7RyZEDqBpyY8xc9zfuaJ8U+wv3g/l6+6nGUpy9DLM6dZaO5Y+THq9fWqh1Q325BawNhIX5UsbIidneD+C4by1rWjKSivZdabv/P4N/vRVdd3+dzZpVU88c0BxkT4cOsU61teoMcThhBiPlpj+LXSMGpQSlkrpSw2PN+B1iDeJ1pejQmjJXbCjpEBI5tW4KtvrOfBjQ/SKBtZOGVhq20VjnaOXDXkKlbMXEFCYALPbX2Ov/70V46XH281DtXg3f1yyqpJy69k6pBAS4eimMGM+P6su38K88ZH8OnWo0xbuJFvduV0epbfRr3k71/sQQIvzxlllZ0fTEoYQohoIYSz4flUIcTdQogOl8GEENOB/wNmSimrmm0PEELYG55HAYOBwx09f29kXDipNaMDR5Opy0RXq2PRzkXsLdrLMxOeYYBX+9Vx/T368/Z5b/PMhGdIKUnhim+v4NNDn3Ky/uQZ+6aXpuMgHIjsF9mlz6OcsiFVm9dp6tDeXxJWWubl4shTM2NZdeckQr1duPfz3Vz7/lb2Zpeh13cscbz322G2HSnhqZmxVtut2tRy8nIgSQgxCHgXWAl8BlzU2gFCiGXAVMBfCJENPInWK8oZWGtojDV2n50MPCOEqAf0wK1SypJOfaJepKq+itLa0lZLGACjArWOY4t2LGJ5+nLmxszlgogLTL6GEILZg2czPmQ8T29+mue3Pc/z256nv3t/oryjGNRvENHe0STnJxPRLwIne9ubBsFSNqQWNk2Yp9i2uNB+fH37RD7bdowXf0xh5hu/4+PmyLhIP86K8mV8tD+DAz2aulP/2YFcHQvXpDIjLpgrRnfPnFrmYGrC0Bu6ws4GXpdSvm5ooG6VlHJuC5s/aGXf5WhJqU9pq4eUUZx/HA52DixPX06sXywPJD3QqWsFuwfz5rQ32Zy7mQPFB8jUZZJZlknyiWRqG2sBuCTqkk6dWzlTXYO2GNFlCaEWW5td6Vn2dlrnhovj+/NzSgFbDhezObOYHw+cAMDX3YmRYf3wcHFsmh7ezfCzYlcOPm5OPDc73qr/v5iaMOqFEHOBeWgD7wB6ZoIjG9Z84aTWuDq4EucXR2ZZJi9NealLJQAhBBNCJzAh9FRPqEZ9IzmVORzRHSHWP7bT51ZOl5xVwsm6RqYOUdVRfY2vuxNXJoZxZaLWO/F4SZWWPA4Xk3pCW/ipqq6BqrpGqusaadBLXBzteO+GpA5NLGkJpiaMG4FbgWellEeEEJHAx+YLq28wDtprq4QB8K9J/6K+sd6k7rEdZW9nzwCvASa1iSimM3annTDI39KhKBYW7utGuK8bVyWFt/h+XYMevZRmnYSxu5iUMKSUB4G7m70+ArxgrqD6ipyKHJztnfFzaXsStoFeA3soIqW7bEgtYEyELx6qO63Sju4e+GdObf5vFkLsA1pt6pdSjuj2iPqQ3JO5hHiEWHWdpdJxuYbutMYqCUWxFe19/TG2gt5heDRWQ11HG4lEMU1bYzCU3mtDaiEA5wxV4y8U29JmwpBSHgUQQpxvGIVt9JAQYifwsDmDs3W5lbnE+fXM1OVKz9mQWlKYR1oAACAASURBVKC60yo2ydTKMyGEmNjsxYQOHKu04GT9Scpqy1QJw8bUNej5PaOIKUMDVFWjYnNMbZG7CfhQCGGcCavMsE3pJFPGYCi9T/JR1Z1WsV3tJgwhhB0wSEo50pgwpJQ6s0dm40ztUqv0LhtTVXdaxXa1W60kpdSjzf+ElFKnkkX3aGvhJKX3+kV1p1VsmKntEOuEEA8IIcKFEL7GH7NGZuNyK3NxsXfB10X9Gm2FsTutmmxQsVWmfg36i+HxjmbbJBDVveH0HbmVagyGrTF2p52qutMqNsrUkd59fs7rTw5+gqOdI3+J+Uv7O5tAjcGwPRtSCwjp58Jg1Z1WsVEmV7QKIeKA4YCLcZuUcqk5grI2uZW5vJT8Eo2ykWD3YKaET+n6OU/mMiJADZS3FSUn6/g9o4iZo9TstIrtMnUBpSeB1w0/5wAvAjPNGJdVWXpwKQJBdL9oHt30aFOX2M6qrKtEV6tTPaRsgF4v+WL7cc5duIHaBj2zE9S/qWK7TG30vhKYBpyQUt4IjASsa3VyMymrKePr9K+5KOoiXjv3NRplIw9ufJD6xs6v36t6SNmGtPwKrn53C/+3fC+DAz34/p6zGRupOjEotsvUhFFt6F7bIITwAgqAlufqtTHLUpdR3VDNjbE3MsBrAE9PeJq9RXtZtHNRm8ftLdzLwuSFVNRVnPGeGrTXu1XXNfLCjylc9OpvpBVU8OIVI/h8wXiGBHlaOjRFMStT2zCSDWt4vwfsACqBze0dJIRYjDaBYYGUMs6wzRf4HIgAsoA5UspSoVX8voq27GsVMF9KubNDn6abVTdUs+zQMqaETWGQzyAALoy4kOQTyXx88GMSgxKZNmDaacfUNNTw393/ZenBpeilnh35O3j7/LfxcvJq2seUhZMU63Sk6CTXf7CV7NJqrhgdxqMXxeDn4WzpsBSlR5hUwpBS3i6lLJNSvg2cD8wzVE21Zwkw/U/bHgbWSykHA+s5NYHhDGCw4WcB8JYpsZnTivQVlNaWcmPc6R/1wTEPMtxvOI9vepzsiuym7bsLdnPVt1ex5MASLh98Oc+f/TyHSg5xy5pbKK8rb9ovpzIHVwdXfJx9euyzKN3jqVUH0FXXs+zms1g4Z6RKFkqfYmqj98dCiJuFEDFSyiwp5V5TjpNS/gqU/GnzZcBHhucfAbOabV8qNVsAbyFEf1OuYw4N+gaWHlzKyICRjA4cfdp7TvZOvDTlJQAe2PgA5XXlvLj9RW744QbqGut49/x3eXL8k1wcdTGLpi4ipTSFBWsWoKvVBsnnVuYS6qF60/Q2G9MK2ZhWyN3nDmZ8dNuLXimKLTK1DWMx0B94XQhxWAixXAhxTyevGSSlzDM8PwEEGZ6HAseb7Zdt2HYaIcQCIUSyECK5sLCwkyG0b03WGnIqc7gp7qYWb+zhnuH8c+I/OVB8gPO+PI+PD37MnKFz+PqyrxkfMr5pv6nhU3ll6iuklaaxYK2WNNQYjN6noVHPs98dZICvGzdMUCsgKn2TqVVSvwDPAo+jtWMkAbd19eJSSkkHF2KSUr4rpUySUiYFBJhnCgYpJYv3LyaqXxRTw6e2ut+0gdO4Of5m+rv354MLPuAfZ/0Dd0f3M/abEj6FV855hfTSdG5eczPZFdmEuKuE0Zt8kZxNWn4lj8yIwdnB+tdeVhRzMLVKaj3wO9oUIanAGCllTCevmW+sajI8Fhi253B6z6sww7Ye90fuH6SWpjI/dj52ou1f0d2j72blrJWM7T+2zf0mh03mlXNeIaMsg8r6StVDqhepqKnn5bWpjInwYXpcsKXDURSLMbVKai9QB8QBI4A4IYRrJ6+5CphneD4PWNls+w1Ccxaga1Z11aMW719MoGsgl0Rd0v7OHTA5bDKvnvMqXk5exPrHduu5FfN5e2MmRZV1PHbxcNXupPRpps4ldR+AEMITmA98CAQDbXYREUIsA6YC/kKIbOBJ4HngCyHEX4GjwBzD7t+jdanNQOtWa0ovrG63v2g/205s4/7E+3G0d+z2858ddjabrt6kbjy9RHZpFe/9doRZo0IYFe5t6XAUxaJMShhCiDuBs4FEtLETi4Hf2jtOSjm3lbem/XmDoT3jjhb27VGL9y/G09GTK4dcabZrqGTRe/znp1QE8OD0ztbAKortMHXgngvwMrBDStlgxngsqkHfwM/HfmZuzFw8nNSMo33d7uNlrNydyx3nRBPq3dkaWEWxHab2knoJcASuBxBCBAghbG7K8/K6chplI+GefWLWE6UNUkr+tfog/h5O3DZ1kKXDURSr0JHZah8CHjFscgQ+MVdQllJWWwaAt7Oqq+7rvtuXR/LRUu6/YKhablVRDEztJTUbbTrzkwBSylzA5mZaM47E7ufcJybiVVpxQlfD49/sJzbEizlJqrSpKEamJoy65oPshBBnjk6zAWU1qoTR1zXqJff8bxe1DXpem5uAvZ3qoKAoRqYmjC+EEO+gze90M7AOeN98YVmGsUpKlTD6rv/+ksHWIyU8PTOW6ADV8UFRmjN1HMZLQojzgXJgKPCElHKtWSOzAOOMsqqE0TdtzyrhlXVpzBoVwpWJYZYOR1GsjsmteYYEsRZACGEnhLhWSvmp2SKzgLLaMhyEQ4vzQSm2rayqjnuW7SLc141/zY5XY2UUpQVtVkkJIbyEEI8IId4QQlxgmLbjTuAwp0Zo24yy2jL6OfdTN4s+RkrJQ8v3UlhZy+tzE1SvKEVpRXt/GR8DpWir6/0NeBQQwCwp5W4zx9bjdLU61X7RB32y5Sg/HcjnsYuGMSJMVUcqSmvaSxhRUsp4ACHE+0AeMEBKWWP2yCygrLZMtV/0MYfyyvnnd4eYMiSAv06yubGoitKt2uslVW98IqVsBLJtNVnAqSoppW8oq6rj9k930s/VkYVzRmKnutAqSpvaK2GMFEIYF6MWgKvhtUCbL9DLrNH1MF2tjji/OEuHofSAugY9t32yk5zSaj69eRz+am1uRWlXmwlDStmnlhbT1epUlVQfIKXk8W/2s/lwMYv+MpIxEb6WDklRegVTB+7ZvOqGamoba1WVVB/w7q+H+Tz5OHedO4jZCWq8haKYSiUMAzWPVN/w04ETPP9jChfH9+e+84ZYOhxF6VVUwjBQM9Xavv05Ou79325GhHmrRm5F6YQeH6EkhBgKfN5sUxTwBOAN3AwUGrY/KqX8vqfiUvNI2bYTuhr++tF2fNwcee+GRFwc+1TznKJ0ix5PGFLKVGAUgBDCHsgBVqCt4b3IsFhTj1MlDNvV0KhnwcfJVNY08NVtEwj0dLF0SIrSK1m6SmoakCmlPGrhOCivVRMP2qpPthxlb7aOF68cybD+NtUTXFF6lKUTxtXAsmav7xRC7BVCLBZC+LR0gBBigRAiWQiRXFhY2NIunaKqpGxTyck6Xl6bxqRB/lwUH2zpcBSlV7NYwhBCOKGt4velYdNbQDRadVUesLCl46SU70opk6SUSQEBAd0WT1ltGa4OrjjZO3XbORXLW7gmlZN1jTx56XA1qaSidJElSxgzgJ1SynwAKWW+lLJRSqkH3gPG9mQwatCe7TmQq+Ozbce4YfxABgfZ3IrCitLjLJkw5tKsOkoI0b/Ze7OB/T0ZjJp40LZIKXl61UF83Jy4V423UJRuYZGJ/w1rgp8P3NJs84tCiFFo64Zn/ek9s1MTD9qW1Xvz2JZVwnOz4+nn6mjpcBTFJlgkYUgpTwJ+f9p2vSViMSqvLSfEPcSSISjdpKqugee+P0RsiBd/GRNu6XAUxWaopcUMVAnDdry9IZM8XQ2vzU3AXo3mVpRuY+lutVZBL/WU15WrhGEDjpdU8favh5k5MkTNQqso3UwlDKCirgK91KtGbxvw7HeHsBeCRy6KsXQoimJzVMJATQtiC/R6yctr0/jxwAlunxpN/36ulg5JUWyOasNAjfLu7arrGnngqz18tzePqxLDuHVqtKVDUhSbpBIGp9bCUCWM3ie/vIablyazL0fHoxfFcPPZUWpEt6KYiUoYqITRW+3P0fG3j5Ipr6nn3euTOH94kKVD6j1KjkB1CYQmdv1cmb9A/5HgpjoZ2DrVhoGqkuqNftiXx5Vv/4G9nWD5bRNUsuio1ffCp3NAr+/aeSpOwMezYMPz3ROXYtVUwkBLGHbCDk8nNd+QtdNV1/Pkyv3c9ulOhvX34ps7JqopyzuqtgKyfoeqIig40LVzZf6iPab+AFJ2PTbFqqkqKbQqKS8nL+yEyp/WSq+XLN+ZzfM/pFBaVcf8CRE8PCNGrZzXGUd+BX299vzwRgiO7/y5Mtdrj7pjUHAQgmK7Hp9itdQdEjXxoLU7kKvjqnc28+BXexno58a3d03iqZmxKll0VvoacPIEn0g4srHz59HrtRJG5GTtdeoP3ROfYrVUCQM1LYi10lXXs2htGks3Z+Hj5sR/rhzBFaPDsFPTfZypMBV8o8G+nT9pKSF9HURNAc9g2L0MGuvBvhMTNObv06q1Rl2rVXOl/QiTH+hc/LakugwaasHT9trVVAkDbeJBVcKwLhtSC7hg0UaWbs7iurMG8vP9U7kqKVwli5ZU5MNbE+D3V9rft+AQlGfD4AsgcgrUn4Ts5M5dN/Nn7TFqKgy9SDtPZUHnzmUrGhtgycXw1njQ5Vg6mm6nEgaqhGFNKmrqeXj5XuZ/uB0vF0dW3D6RZy6Lo5+bmqK8VTnJoG+AXR+33+spY632OOg8iJgEiM5XS2Wsh6A4raQyZDogIe2nzp3LVmx7B/L3Q005LP+rlkBsiEoYqIRhLf7IKGL6K7/xRfJxbpkSxbd3TWJkuCr5tStnp/ZYmgXH/mh73/S12k2+X6g2bqL/SK3hu6PqTsKxLRB9jvY6OB68wrRqqb6qPA9++bdWepv1FhzbDD//09JRdas+nzDqGuuobqhWVVIWdLK2gSdW7uea97fi5GDHl7dO4JEZw1Sjtqlyd4LfIK0he/dnre9XU67dxAaff2pb1BTI3q4lgI7I+l3raRV9rvZaCBhyoVZNVV/T8c9gC9Y+Do11MOMFGHEVJM7XqgltqNTV5xOGmnjQMnRV9azak8s9/9vFhOd/5uMtR7lpYiTf3302iQN9LB1e7yGlVsIYOAHiZsOBb6C2suV9D2/Qqq4GNUsYkVO0G//RzR27bubP4OACAyac2jZ0BtRXQdZvHf4Yvd6RX2HflzDpXvCN0rZNfx6C4mHFLVB23LLxdROL9ZISQmQBFUAj0CClTBJC+AKfAxFoy7TOkVKWmjMONcq75xwtPsmaA/msO5RP8tFSGvUSP3cnzh8exNVjwklS61d0XMlhqCmDkNEQOAx2LoWDKyHh2jP3TV8Dzv0gfOypbQPGg70THNkAg88z/bqZP8PAieDocmpbxNng6A6p359eiumIhlpwcO7csZbSWA/fPQDeA2HSfae2O7rCnI/gnSnw1U1w4/ed641mRSzdrfYcKWVRs9cPA+ullM8LIR42vH7InAGoeaTMr6KmnoVrtO6xegkxwZ7cNiWac4cFMjLMW62K1xW5u7TH0EStHcE3GnZ/embCkBIy1kH01NNvWk5uED5OK32YSpcNRakw+obTtzu6aG0aaT9p1+voJJCHVsPXC+COreDdi5bW3fKm9vuY+7mWJJrzi4aZr8FXN8K6p+DCZy0SYnextiqpy4CPDM8/AmaZ+4IqYZiPlJLVe3OZtnAjH23Wusdueugcfrx3Mg9cOJTRA3xUsuiqnJ1a1VDgMO0GPeoaOPq7VvJoLn8/VORpDbJ/FjkFTuyDk8WmXdPYndbYftHc0BlQngMn9nbscwAc+lbr5ttWO4y10eXAhhe0bsVDp7e8T9zlMOZvsPkNSPm+Z+PrZpZMGBJYI4TYIYRYYNgWJKXMMzw/AZwx8kUIsUAIkSyESC4sLOxyEKpKyjyOFp9k3ofbufOzXQR6OfONoXtsmI+bpUOzLTk7IHjEqVLDyLmA0AbkNZferDvtn0VN0R6zfjXtmpk/g2d/LUn92eALteundrC3lJSnuvfu/rTrkyL2lDWPgWyE6f9ue78Ln9N6pH1zq9abrZeyZMKYJKUcDcwA7hBCTG7+ppRSoiUV/rT9XSllkpQyKSAgoMtBqITRveoa9Ly+Pp0LFv3KzqOlPHXpcFbeobrHmkVjA+TtOX2K8n6hWrXQnmWn33TT12qJxTP4zPOEjNZ6WJnSvVbfqFVfRZ/bcpWTRwCEJWntGB1RlK6VgAZOgrKj7XcPtgaZv8CBFXD2/eAT0fa+Ds5w1UdaYvzyRmio69q1TxZZJKlaLGFIKXMMjwXACmAskC+E6A9geDT7sFFdrQ5ne2dcHdSSnl11IFfHzDc2sXBtGucPD2L9/VOYPzFSVTuZS2EKNFRD6OjTt4+6FnTHT5UYqsvg+NbWG6LtHSBiomkD+HJ3Q3Vpy9VRRkNnQN5ubVyCqYxtKBf9p/3uwdagMA1W3a31iJpwt2nH+EbCZf/VukGvfbzz126sh49nw9d/6/w5OskiCUMI4S6E8DQ+By4A9gOrgHmG3eYBK80dixq013X1jXpeW5/OZW/8TvHJOt67IYk3rhlNkJdL+wcrnZdrGLAX8qeEEXOx1hvKeNM9/ItWbdJS+4VR1FSt3aPsWNvXbD4dSGuGzNAeOzKI78hG8B4AQcPb7x7cWbm7YNk18N65UFfV+fOkr4P3z9OS9RUfnN5TrD3DZ8K422Dr21pvts747WWtjSh2dueO7wJLlTCCgE1CiD3ANuA7KeWPwPPA+UKIdOA8w2uzUjPVdk3qiQouf/MPXl6bxsUj+rPm3slqMaOekrNTSwzGfv9Gjq4QfwUcXAU1Ou0G59IPQpNaP1ekoR2jvWqpzJ+1unh3/9b3CRym3fxNTRj6Rm3shjGGUddqjd+dvaH+Wc5O+Owv8O5UyNqktfv8trDj55ESNr8Jn12lfb6bfzmzdGeK85/RqhFX3nlm54T25O2FX1+E+Ktg2KUdv3YXWSRhSCkPSylHGn5ipZTPGrYXSymnSSkHSynPk1KWmDsWNfFg5zQ06nlzQwaXvr6J3LJq3rp2NK9enYCPu5OlQ+s7cnZAaALYtfBnPOpa7Rvw/q+1+aOip7U9k23gMHAPbLtaqqYcsrdp52qLEFop4/AG077J5+3WElvUVO11+LhT3YO7ImeHtqrge+do05ic8w+4bx+MuBr+eA2KMkw/V0MdfHs3/PSI1iPqph873/XXwQmu/FD7PX053/SR8Q118M3t4OoLM17s3LW7yNq61fY4VSXVMXq95Ns9uVzwyq+8+GMq04YFsua+ycyI72/p0PqW+hptwaI/V0cZhSaC/1Bt6dTK/Laro0C7eUVO1kYst7ZyXtYmbaR4W+0XRkOnQ0ONaeM7jKUaYwmjre7BplrzuFb1lL0Nzn0c7t0HUx7USlrnP6N1Rf7hQdNWCTxZBEsv0wZFTv4/mPMxOHt0Li4jn4Ew622t08Kax0w75reXtCnlL33FYuunq4ShEoZJpJSsO5jPxa9v4q5lu3CwE7x7fSJvXjsaP49eNjLXFpzYp928W6sSMd50K09orwe1UyoArXttZb7WmN6SzPXaSO7mI8VbM3CS1nid+l37+x7ZCIGxWg8ro5FzQdid2T3YFOW52mC6uCu1RDH5AXBptoyvZxCc85hWvXZoVdvnKjmsJZ7cnVp7xbmPtVyi64yYi2D8nbD9fdi/vO19c3dr1WgjrtbaqCykTycMKaWqkjLBHxlFXP7WH/xtaTJVdQ288pdR/HDPZC6IDUZ0dDSv0j1aa/BubuTVIOwhJAE8Ats/Z3vtGJk/a1OimzJ1h4MTDL9MqxKrLmt9v/oarbrIOBbEqF8oRLXQPdgUyYu1dpFz/wHOni3vM+Zv2jxPPz7SeuN6yWFYcgnUlsP87yH+yo7FYYrznoKwsVqPqx1LWu5u21CrVUW5+cMMszfrtqlPJ4zK+koaZINKGM1IKTlcWMmXycd5ePleznt5I9e8v5UTuhr+fXk86/4+hVkJoaqrrKXl7ACPYPAKaX0fz2BtKoqpj5p2Tp+B2niCIxu1m3TpUW2aj99fha9v0W6gplRHGY1boE1GuOuT1vc5vlWruoqccuZ7o645vXuwKeprIPlDrZ3BN7L1/ewd4OKF2qj0X/9z5vvFmfDhxVBfDfO+hbDEM/fpDvaOcNUSCBgK394Drydq8TdPHBtfhIIDcOmr4GrZiTktPZeURalBe5r6Rj1fJB/nl5RCdh4rpeSk9p/Vy8WBxIE+3DB+IHOSwtV049YkZ6dWHdVeCe+s2zp23qip2g3+32FaTyUjj2BtlHhHunL2H6nNZrvtHS0Ouxb+/xzZqJWCBk44872YS051D46aato19y/Xlo0dd0v7+w4YB6Ou06bsGHWNdtMGLVksuQQaa7VkERxn2rU7q18o/G29NtfXhudh9b1a9dPZf9eq6jYt0joxtDb1SA/q0wmjvLYc6LvzSEkpWX+ogOe+P8ThopNE+LlxbkwgSQN9SBzoQ3SAh1oS1RrV6KA4HUb8pfvPnThfW/LVJ0K7gQYOA/8hnW9kPetW+OIGrYttS3XvhzdqDfTN2xiMHF207sG7l2kD+lza+WInpTa+IXC41oBvivOegpRv4fsH4IZVWrL46BJtXYt530JQrGnn6SohtIGVg87T2oo2PA+rDTPfeoZoU4tYgT6dMPryWhiH8sr513cH+T2jmKgAdxbPT+KcoYGqTaI3yN2tPXZmDEB7QhLgmv913/mGXqytxLf17TMTRo1Oa4s5+4HWjx91ndYmceAbSJzX+n6gtYWc2KtV3Zj6/9gjAKY9Ad/dr/VC2va+tj5ITyaL5oTQkkb0NK3NaPsHMP52cLWOe5RKGICXcwvfbmxUUWUtC9ek8fn2Y3i6OPLUpcO59qyBONr36eas3iVnh/YYkmDZOExh7wBj/6ZN7Z1/4PSbcNbvIPVnNng3Fzpa6x68+9P2E8bWt8DFG+LndCzGxBth58fw87+0huV5q7UR55YkhNazzZTebT1IJQxsu4Sh10sOnShnU3oRmzKK2HakhEa9ZN6ECO6ZNhhvNzXQrtfJ3Qk+kRbri99ho+dpU4BvfUdbG8Lo8AZwcIWwMa0fK4S2tsfaJ+DE/tbbE8qOa+tpTLhTW+OjI+zsYebrsO5JreqnpVl4FaCPJwzjWhheTrZVwsgureL3jCI2ZRTzR0YRxYZG7CFBHlwzbgDXnTWQ6IAuDjxSLCdnl9Zg21u4+cKIObD3C63NwJjojmyEgePb76Y78hrY9Ap8NkerKvKLPnOf5A8AqXWX7Yz+I+D6FZ07tg/p0wmjrLYMTydPHOx696+hrKqOzZnFbMoo4veMIrKKtekYAj2dmTIkgImD/Jk02F9NBmgLKvKhPBtCb7d0JB0z7hbY+ZH2M+k+qDihDRAcObf9Yz0CYN4q+Gim1ntp/urTk0ZdlTaGIeYSbY4nxWx6952yi3S1ul5ZHVV6so7tWSUkHy1ly+Fi9uXokBI8nB0YF+nLDeMjmDjInyFBHqoR29aYMmDPGgXFamt+b3sfxt+lTUECbbdfNBccr5Uuls6EJRfD/O9OJY19X2pTro+71TyxK01UwugFCSNPV82Ww8VsO1JKclYJ6QXayFQneztGhXtz77QhTBrsx4gwb9V4betydmpTZvQfYelIOu6s2+B/12jThRzeqDVQB3fgcwTHaUnjo0u1pDHPUNLY+o42arulsRxKt+rTCaOstgxvF+tLGHUNepKzStiYVsiG1EJS8ysA8HR2IDHCh1kJoYyN9CU+tJ8aTNfX5O7Uxhk4uVs6ko4bMl2rMtrytrbuRuTklgfztSUoVksUxqQx9SFtFPTMN0zvSqt0Wp9PGBH9IiwdBgC6qnq+35/H+kMFbM4s4mRdI472gjERvjyaGMPEQf7EBHupKTn6Mim1EoYFJ5/rEjt7GLsA1vxDe332fZ07T9BwrR3jo0u1wW1uftr6EIrZ9emEYekqqdqGRn5JKWTFrmx+SSmkrlFPqLcrsxJCmTo0kPHRfng49+l/IqW5sqNQXWKeAXs9JeF6+OU5bY6pyKmdP0/gMK2k8elVMPbmjq16p3Raj9+NhBDhwFK0Vfck8K6U8lUhxFPAzUChYddHpZQdXEnedPX6eirrK3t8HqmTtQ3szdaxak8u3+3NpbymAX8PZ647ayCzE0KJC/VSDdVKyw6t1h7Dz7JsHF3h6q11fc1Y13L32I4IjIF796qqqB5kia+vDcD9UsqdhnW9dwgh1hreWySlfKkngjCOwTBXCaNRLzlSVMmhvApST1SQcqKC1PxyjpdUA+DqaM+FsUHMHh3GxGg/HFRjtdKWhjrY/F+tp5GlRyF31fnPwHlPd8+NXiWLHtXjCUNKmQfkGZ5XCCEOAaE9HUd3TzxYUVPP7uNl7Dhayo6jpew+VkZFbQMA9naCSH93RoR5MycxnJj+XkyI9sNdVTcpptr/FVTkaiOSezsh1I2+l7LoHUsIEQEkAFuBicCdQogbgGS0UkhpC8csABYADBjQ+UE6TVObO3WuSkpXXc+Ww8Vszixmy+FiUvMrkFL7Oxga5MnMUSEkDPBhWH9PogM8VG8mpfP0em1NiqB4q5tbSOlbLJYwhBAewHLgXilluRDiLeCfaO0a/wQWAjf9+Tgp5bvAuwBJSUkmLMjbsqaE0d6UyWjdXCtq6jmYV87vGcX8kVnE/hwdeqlVLSVF+DA9LpjEgT6MCvfG08Wxs2EpypnSf9JGRV/+vvpmrliURRKGEMIRLVl8KqX8GkBKmd/s/feA1eaMoaU2jEa95OlvD7AnW0dFTT0VNQ1U1NRTU39qiUgHO0HCAG/uOncwEwf5MyrcGycH1f6gmNGmV6DfgI4tXqQoZmCJXlIC+AA4JKV8udn2/ob2DYDZwH5zxtHSTLVfJB9n6eajjI30Jay/F14uDni6OOLp7ICniwMR/u6MifBVbQ9Kr1qsOQAADIhJREFUzzm2BY5vgRkvalOFK4oFWeJ/4ETgemCfEMKwEgyPAnOFEKPQqqSyABPWWOy8stoyHOwccHPQpkIur6nnpZ9SGRPhw+cLzlJdWxXrsOkVcPWFhOssHYmiWKSX1Cagpbux2cZctMQ4aM+YGF5fn05JVR1LLhmrkoXSvWp04OgG9h1s2yo4BGk/wNRHeudUIIrN6bOV77paXVMPqcOFlSz5I4urEsOID+vZgXyKjcveAYvi4a0JcHx7x479/TUt0YxdYJ7YFKWD+mzCKKstaxrl/ex3h3B2sOeBC4daOCrFpmQnw8ezwLWftmbD4gu0eZTqq9s/VpcD+77QptLoLSvrKTavTycMb2dvNqYVsj6lgDvPHUSgp5qPRukmx7fDx7O1m/2NP8Dtm2H0DfDH6/D2JDi2te3jt7ypTTY4/o6eiVdRTNBnE4auVoeXUz/+ufogA/3cuHFihKVDUmzF8W2GZOEH87+HfmHg4gWXvgrXf6NN87H4QvjpMW3hn7qq038qTmgryMVdAT4DLf1pFKVJn+ynJ6WkrLaMnGJBRkEl716fiLODGomtdINjW+GTKwzLiq6Gfn+a9Sb6HLj9D1j7JGx+Q/tpzcR7zBuronRQn0wY1Q3V1OvrST5cy8RBfpw/PMjSISm24NhW+ORy8AjS1mvwCml5P2dPuORlGDFHG2fREp+B2gpzimJF+mTCMI7yrql15om5saobrdKy+hooztCm5ShM0bq5FqZq61JI/Zn76xvAN7rtZNHcgLO0H0XpJfpkwjiQon2ru9d7H0PdKgFPywakdE19NWSsh4PfwNHNMHQGTLrvzOqgto4vSteSQaEhKRSmQMnhU4lB2INvFAQM1c7f0pgKe2dInAeewd332RTFivTJhBHkHcSFNf5ML94Er47U/sgn3Wfat0LFOtRXQ/paLUmk/QR1ldqI6NBE2PEh7PxI65U06T6t0fnPijO1Yw+uhBP7Tk8MftHautmxsyEgRlvdzW8QODj37GdUFCsjpOz0hK8Wl5SUJJOTkzt/gpIj8NtC2LMMhB2Mntexb6ZK62oroShV+7ZurMopTtd6DgXEGG7Ehkev0NZnYZVS6zVk/OZvPNeJvdoyn66+MOxSiJ2lLS5k7wilR2HTy7DrE+3fNeF6OPvvWhXTwRVwYCXk79POHzYGos7RkkJAjCExOPXc70lRLEAIsUNKmdTh4/p0wjAqzYLfXobdn2o3mLAxWtVDwLBTNzX3AG3fihOGOu1m1Re1leA/2HDTGart7xul3bzqqqAo7VQ9eGGqlqg8g5vtb3h07eRiTjW6U9UoxptqaRbIxq7/bjqjoU5b7MfI3gn8h2jf3KtKtDhPFp5638kT3P1aPld1qfb5jFx9tN9XcBwMvciQJFopKJcd0/5dd32i/S6MpYjwcTB8Fgyf2XLpQ1FsnEoY3aH0KGx9G3J2QEEK1Da/UflqN52Wbl7OHqcaQ43sHLUkU5GHNp+iYZvfIPCNhP9v7/5jrizrOI6/P/wQmaD81JxgylomFCEyV9M1Jn8o4LRVm6YtVm3OtPVrrTBd4Z/RHxnVZo5hukw0LbbUSiqbtVKE+CGYxAOSShSgAvkjf8C3P67vgfvBBzjPwwP34ZzPazs71/ne5z7n+j7PfZ/rXPd1n+ve/a/SkLz12v51hr2r943G/3Z3/3AeNBTGvjcbrJq+KQ8YVHIcm9/aR571zg/1V1+sNKLPdP+7Vg0Znj2Sc/Y33L09SWHn87B8EQw7Fc693D1I63huMPrbvkMhlQ81DXhnr6P64fXmq9mbyG/7u7eWD+7G8xu9joa9e2HXc917B2++0rt6Dj6pNBCNwzwjzoQB/k2JmR2cGwwzM2tKXxuMjp0axMzMescNhpmZNcUNhpmZNaXlGgxJl0paL6lL0ty662NmZkVLNRiSBgI/AmYCEynX+Z5Yb63MzAxarMEALgC6ImJTRLwJLAauqLlOZmZG6zUYZwDPVx6/kLF9JF0rabmk5du3b8fMzI6NVmswDisibo+IaRExbezYsXVXx8ysY7TabLVbgPGVx+My1qMVK1bskPTPgy1vwhhgxxGsf7xy3p3FeXeWZvLu07V/W+qX3pIGAf8AZlAaiieBqyNi3VF6v+V9+bXj8c55dxbn3VmOZt4t1cOIiLclfQH4LTAQWHS0GgszM+udlmowACLiYeDhuuthZmbdHXeD3v3s9rorUBPn3Vmcd2c5anm31BiGmZm1rk7vYZiZWZPcYJiZWVM6ssFohwkOJS2StE3S2kpslKSlkjbk/ciMS9KCzHeNpKmVdebk8zdImlOJny/pqVxngdTb66IeHZLGS3pU0tOS1kn6UsbbOndJJ0paJml15n1Lxs+W9ETW9V5JJ2R8SD7uyuVnVV7rxoyvl3RJJd6y+4WkgZJWSnowH7d93pI253a4StLyjNW7nUdER90op+tuBCYAJwCrgYl116sPeXwEmAqsrcTmA3OzPBf4TpZnAb8GBHwIeCLjo4BNeT8yyyNz2bJ8rnLdmXXnnPU6HZia5eGU3+1MbPfcsy7DsjwYeCLreB9wVcZvAz6f5euB27J8FXBvlifmNj8EODv3hYGtvl8AXwV+BjyYj9s+b2AzMOaAWK3beSf2MNpigsOIeAx46YDwFcCdWb4T+GglflcUjwMjJJ0OXAIsjYiXIuJlYClwaS47OSIej7Jl3VV5rVpFxNaI+FuW/wv8nTLfWFvnnvVvXPB9cN4CuBi4P+MH5t34e9wPzMhvkFcAiyPijYh4Fuii7BMtu19IGgfMBhbmY9EBeR9Erdt5JzYYh53g8Dh2WkRszfK/gdOyfLCcDxV/oYd4S8nDDedRvm23fe55WGYVsI2y428EdkbE2/mUal335ZfLdwGj6f3foxXcCnwd2JuPR9MZeQfwiKQVkq7NWK3becv9cM/6R0SEpLY9Z1rSMOAB4MsRsbt6+LVdc4+IPcAUSSOAXwLvq7lKR52ky4BtEbFC0vS663OMXRQRWySdCiyV9Ex1YR3beSf2MHo1weFx5j/Z1STvt2X8YDkfKj6uh3hLkDSY0ljcHRG/yHBH5A4QETuBR4EPUw49NL74Veu6L79cfgrwIr3/e9TtQuBySZsph4suBr5P++dNRGzJ+22ULwgXUPd2XvfAzrG+UXpVmygDX41Brkl116uPuZxF90Hv79J9QGx+lmfTfUBsWewfEHuWMhg2MsujoucBsVl155v1EuV4660HxNs6d2AsMCLLQ4E/AZcBP6f74O/1Wb6B7oO/92V5Et0HfzdRBn5bfr8AprN/0Lut8wZOAoZXyn8BLq17O699I6jpnzGLcnbNRuCmuuvTxxzuAbYCb1GOP36Ocqz298AG4HeVDUOUS99uBJ4CplVe57OUAcAu4DOV+DRgba7zQ3JWgLpvwEWUY7trgFV5m9XuuQOTgZWZ91rgWxmfkDt+V36IDsn4ifm4K5dPqLzWTZnbeipnxrT6fkH3BqOt8878VudtXaNedW/nnhrEzMya0oljGGZm1gduMMzMrCluMMzMrCluMMzMrCluMMzMrCluMMwASXtyVtDG7ZCzlkq6TtKn++F9N0sac6SvY3Ys+LRaM0DSKxExrIb33Uw5Z37HsX5vs95yD8PsELIHMD+vG7BM0nsyPk/S17L8RZXrc6yRtDhjoyQtydjjkiZnfLSkR1SuabGQ8oOrxnt9Kt9jlaQf52SDAyX9RNLarMNXavgzmAFuMMwahh5wSOrKyrJdEfEByq9hb+1h3bnAeRExGbguY7cAKzP2Tcp0JgDfBv4cEZMo8wOdCSDpXOBK4MKImALsAa4BpgBnRMT7sw539GPOZr3i2WrNitfzg7on91Tuv9fD8jXA3ZKWAEsydhHwcYCI+EP2LE6mXPjqYxl/SNLL+fwZwPnAkznz7lDKxHK/AiZI+gHwEPBI31M0OzLuYZgdXhyk3DCbMo/PVMoHfl++iAm4MyKm5O2ciJgX5aI3HwT+SOm9LOzDa5v1CzcYZod3ZeX+r9UFkgYA4yPiUeAblOm0h1Fmk70mnzMd2BERu4HHgKszPpMygyiUCeU+kdc+aIyBvDvPoBoQEQ8AN1MaJbNa+JCUWTE0r2bX8JuIaJxaO1LSGuAN4JMHrDcQ+KmkUyi9hAURsVPSPGBRrvcaMCeffwtwj6R1lCmrnwOIiKcl3Uy5wtoAyizENwCvA3dkDODG/kvZrHd8Wq3ZIfi0V7P9fEjKzMya4h6GmZk1xT0MMzNrihsMMzNrihsMMzNrihsMMzNrihsMMzNryv8BntymmbzZJ4MAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(aggr_ep_rewards['ep'], aggr_ep_rewards['avg'], label = 'avg')\n",
    "plt.plot(aggr_ep_rewards['ep'], aggr_ep_rewards['min'], label = 'min')\n",
    "plt.plot(aggr_ep_rewards['ep'], aggr_ep_rewards['max'], label = 'max')\n",
    "plt.legend(loc='upper left')\n",
    "plt.xlabel('Episodes')\n",
    "plt.ylabel('Rewards')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 6. Rendering Test"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "done = False\n",
    "state = env.reset()\n",
    "while not done:\n",
    "    action = np.argmax(q_table[get_discrete_state(state)])\n",
    "    next_state, _, done, _ = env.step(action)\n",
    "    state = next_state\n",
    "    env.render()\n",
    "\n",
    "env.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python(spinningup)",
   "language": "python",
   "name": "spinningup"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
